1 Overview

The IBM System Verification Test (SVT) objective is to execute a set of test scenarios against a test configuration that contains the key requirements and components that will create a load on two Linux SLES 10 - 64 bit machines.

This testing will use test scripts currently used by the Domino System Test Team.

One's perception of system quality is governed under the statement of overall system reliability. A widely accepted definition of software reliability is the probability that a computer system performs its destined purpose without failure over a specified time period within a particular execution environment. This execution environment is known formally as the operational profile, which is defined in terms of sets of possible input values together with their probabilities of occurrence. An operational profile is used to drive a portion of the system testing. Software reliability modeling is therefore applied to data gathered during this phase of testing and then used to predict subsequent failure behavior during actual system operations

A reliability test is one that focuses on the extent to which the feature or system will provide the intended function without failing. The goal of all types of testing is the improvement of the reliability program with specific statements about reliability-specific tests. Reliability is the impact of failures, malfunctions, errors and other defect related problems encountered by customers. Reliability is a measure of the continuous delivery of the correct service (and, the time to failure).

SVT's purpose of running Reliability tests was to ascertain the following:

• Data population for all parts of the infrastructure to force set limits to be achieved and passed
• Running sustained reliability scripts at >100% maximum capacity. Assessing :
• Breakpoints
• System stability pre and post breakpoint
• Serviceability
• Forcing spikes and anti-spikes in usage patterns
• Exposing SMTP, IMAP, POP3 services to 110% of their maximum load
• Flushing out the DB Table spaces to their maximum, proving the maximum, proving ability to recover/get back to a good place when the maximum limits have been exceeded
• Proving serviceability errors and warnings when thresholds are hit

2 Configuration diagram for Wachovia x64 configuration

The Wachovia configuration the domino system test team used, utilized two SLES 10 64 bit  machines.

The environment consists of two Linux servers with 4 partition servers running on each machine. Each partition server will host 1,500 registered users with 1,000 active users. The two servers are clustered and replication is enabled between the two cluster mates. The mail journaling resides on a separate server, each partition server will have an journaling database specific to that server. An agent is used to keep the journaling databases to a manageable level by removing journaled docs after a period of time. Circular transaction logging is enabled for each server and each log resides on a separate disk drive and has its own file system. A dedicated Administration server is used to configure and manage the environment.  It is generally recommended to use separate filesystem and disk for rebuild of database views by update/updall task, if possible.  The design task was run at the start of the test to upgrade the templates. The update task runs for the entire period of the test (until the server is brought down).  However, updall runs between 2:00 AM and 5:00 AM, or until it finishes.

Mail file sizes at the beginning of each week of test runs will be as shown in the table below, with the mail files for active users on each server at ODS 51, the Domino 8.5 On-disk structure. 

Details of System under Test (SUT)

System	IBM System x3850 (domino partitions), IBM System x3650 (journal server)
Processor	4 x Quad Core Intel Xeon Processor x7350 (2.93 GHz 8 MB L2 130w) 4 CPUs Quad Core (domino partitions),  2 CPUs Dual core (journal server)
Memory	6 * 4 GB PC2-5300 CL5 ECC DDR2 SDRAM  RDIMM 24 GB (domino partitions), 8 GB (journal server)
Fiber Channel Adapter	2 2GB adapters on IBM system x3850
Disk Drive	6 * 146GB 10K RPM local disk drives, 32 * 146GB 15K RPM disk drives on IBM DS6800 SAN via  IBM 3534-F08 SAN switch
Operating System	SUSE Linux Enterprise Server 10 (x86_64) SP3
Domino Server	Domino 8.5.1 Gold for SUSE Linux Enterprise Server 10

2.1 Evaluation Criteria
The performance of Domino 8.5.1 is evaluated under the following criteria:
· Server CPU: The overall CPU of the server will be monitored over the course of the experiment. The aim is for the server CPU not to go above 75% over the course of the experiment allowing the server to function appropriately. It is acceptable for the CPU to occasionally spike at this level for a short period of time, but it must return to a lower level. High CPU results from the server being stressed due to processes running such as compact, fixup or replication or from user load or any other third party programs.
· Domino Processes CPU: The previous metric monitors the overall CPU of the server, however, the CPU consumption of Domino specific processes will also be monitored individually. In this manner the CPU consumption of Domino specific processes may be evaluated.
· Server Memory: The server memory metric represents the amount of physical memory available on the server. If the available memory becomes low the server performance could be compromised.
· Server Disk I/O: The disk is a source of contention when a server is under load and performing a high number of read and write operations. The disk queue length is measured to determine if the disk I/O operations are resulting in a bottleneck for the system performance.
· Network I/O: These metrics monitor the network utilization to ensure the bandwidth consumption is acceptable and that the network is not overloaded.
· Response Times from the End-user Perspective: The server response times for user actions represent how long a single user must wait for a given transaction to complete. This metric captures the user experience of the application with the server. At times, response times will be longer when a server is under load.  When response times increase over an extended period, or persist at high levels (e.g. when a database or view takes longer than 30 seconds to open), they indicate that performance indicators are being hit and detailed analysis must be performed to determine the source of the slowdown and seek remediation.
· Open Session Response Times: In addition to monitoring the individual action response times, the Open session response times will also be evaluated in order to ensure the server remains responsive over the course of the experiment.

2.2 Tools

In order to simulate user activity and capture the evaluation metrics discussed in section 2.2 a number of tools must be used.
· Testnsf : The Testnsf tool is the IBM Lotus Domino load generation tool which can be used to measure and characterize various Lotus Domino server capacity and response metrics under load. The load is generated by running workloads that simulate the behavior of Lotus Domino client-to-server operations. The workloads enable simulating consistent, repeatable load against the Lotus Domino server. Testnsf additionally captures action response times as discussed in section 2.2 which may be recorded and analyzed.
· Domino showstats data: The Domino showstats captures important server metrics, a Testnsf client driver may be used to execute the showstats console command at regular intervals for each server in the configuration and will provide Domino-specific data. The resulting data is logged in a text file and may be graphed for analysis.
· Open session: The Wachovia Open session tool measures mail file request/response times. It will open a view of a mail database at a set time interval and record the response time in milliseconds. As a result, a server slow down may be identified by analyzing the resulting response times.
· TPROF: The TPROF tool reports processor usage for individual programs and the system as a whole [1]. This will be used in order to highlight the processor usage of specific Domino processes and can identify Domino processes that are the most CPU intensive.
· NMON: The NMON tool is designed for AIX and Linux performance monitoring and analyzing performance data [2]. NMOM will be used to capture and graph CPU, memory, disk I/O and network I/O metrics for the duration of the test period
2.3 Evaluation Process

The testnsf tool will be used to place load on the Domino server.  In order to simulate realistic load on the Domino server a total of 16 client drivers running testnsf will be used. The testnsf scripts will run 24 hours per day for 7 days.
In order to isolate the performance of the Domino server under load from a single user perspective for standard Notes mail, a client driver will execute a "single user" testnsf script and the Open Session tool.  The results represent a single user experience of how the application will perform at busy times of the day when the server is heavily loaded.
Storage Equipment (IBM DS6800) and SAN Switch (IBM 3534-F08) has been obtained, installed, and configured.  SAN storage is configured as RAID 5.  All Domino databases were located on RAID 5. Domino transaction logs were located on separate disk for each Domino partition in its own file system (part of simple volume with single disk). Operating System, Domino executables were located on separate disk. Paging space was located on a separate disk.
The Domino data directories (located on RAID 5 array) were mounted as directories /notesdata/ms101a, /notesdata/ms102a, /notesdata/ms103b, and /notesdata/ms104b. For this test, view rebuild was created in a separate directory under the Domino data directory. 
A Domino Administration server was created and a large directory (names.nsf) containing over 140,000 users, 46,000 groups nested in 3 hierarchical levels was constructed for an isolated test domain to prevent leakage of test email into the public domain.

2.4 Scenario: Online Mode

The scenario evaluates the performance of Lotus Notes Clients in online mode. Online mode means that the user mail files are stored and maintained on the Domino server. Every time a user performs an action the request is sent to the server and the mail file is modified and updated on the server side.

Table 1 shows the action workload of the built in N85Mail script with modifications to the attachment size. The script reflects the workload that is expected of a single user over the course of a day.

The resulting mail distribution is shown in table 2.

Table 3 shows the action workload of the built in N85DWA script with modifications to the attachment size. The script reflects the workload that is expected of a single user over the course of a day.

The resulting mail distribution is shown in table 4.

3 Test drivers
For the Notes 8.5.1 online mode test, 18 driver systems were used, of which 16 systems were configured as load drivers  (which ran 1000 users), 1 driver ran domino showstats client and OpenSession tool, another was used for administration and monitoring during the script run

The configuration used for the driver systems follows:

Single user load drivers machines:
· Intel Pentium 4, 2 CPUs, 2.4GHz with 1GB memory
· C: Partition (80GB - NTFS) - Microsoft Windows XP Professional SP3 and Notes 8.5.1 gold client

Load drivers and showstats machines:
· Intel Pentium 4, 4 CPUs, 3.0GHz with 2GB memory
· C: Partition (80GB - NTFS) - Microsoft Windows XP Professional SP3 and Notes 8.5.1 gold client, Domino Administration Client 8.5.1

Number of Server Systems
Two IBM x3850 systems with 4 CPUs Quad Core 3.33GHz Intel Xeon processors, 24 GB of Memory for hosting domino mail partitions.  One IBM x3650 system with 2 CPUs 3.0 GHz Intel Xeon processors with 8 GB of Memory for hosting domino journaling server.

The disk configuration used for the system under test follows:

· Single disk drive (146 GB - 10K RPM) - SUSE Linux Enterprise Server 10 (x86_64) SP2, Domino 8.5.1 executables,
· Single disk drive (146 GB - 10K RPM) - Domino transaction logs (each domino partition has its own separate disk and filesystem)
· Single disk drive (146 GB - 10K RPM) - Swap space
· 32 * 146 GB 15K RPM drives for Domino data and user mail files
· /notesdata/ms101a, /notesdata/ms102a, /notesdata/ms103a, /notesdata/ms104a, /notesdata/ms101b, /notesdata/ms102b, /notesdata/ms103b, /notesdata/ms104b, /ms10*a/mail1, /ms10*a/mail2, /ms10*b/mail1, /ms10*b/mail2  located on 32 drives RAID 5 array.
· 8 mail-in db's for each domino partition were created on domino journaling server to journal all messages.
· The RAID array was connected to SUT through the IBM 3534-F08 SAN switch via 2 2GB Fiber Channel adapters.

Software Versions:
Software versions used on the system under test were as follows:

· SUSE Linux Enterprise Server 10 (x86_64) SP3
· Lotus Domino 8.5.1 Gold for SUSE Linux

Software versions used on the client drivers machines were as follows:
· Microsoft Windows XP Professional Version 2002 SP2
· Lotus Notes 8.5.1 Gold and Domino Administration 8.5.1 client Gold for Microsoft

Conclusion and Summary
The test results demonstrate that the IBM System x3850 configured as described in this report can support up to 8000 concurrent, active Notes 8.5.1 users with an average response time well below 2 seconds.
These results are based on running x3850 system as a dedicated Domino server in four partition mode.  The addition of other application workloads will affect the number of users supported as well as the response time.  Achieving optimum performance in a customer environment is highly dependent upon selecting adequate processor power, memory and disk storage as well as balancing the configuration of that hardware and appropriately tuning the operating system and Domino software.

4 Configuration settings

Server Notes.ini

The following notes.ini variable was added to each of the Domino Servers:

CREATE_R85_DATABASES=1 (to enable the creation of Transaction Logs in 85 format)
MEM_AddressableMemSizeMB=3500  (the amount of memory Domino will see as "addressable" memory).
ConstrainedSHMSizeMB=3000 (enables the constraining of shared memory only)
NSF_buffer_pool_size_MB=512 (set the size of the UBM – Unified Buffer Manager)
HTTPJVMMaxHeapSize=256M (set the maximum size of HTTP JVM)